TCP, defined for example in RFC 793, is a widely used protocol of the Internet that allows for reliable and ordered delivery of data. For example, web browsers commonly use TCP when connecting to servers on the Internet. The TCP segment includes a header that includes a number of fields including source port, destination port, sequence number, acknowledgement number, data offset, reserved, control bits, window, checksum, urgent pointer, options, padding, and a field for the data. The TCP segment is commonly encapsulated into an IP or IPv6 packet whose header includes a number of fields including among others source IP address, destination IP address, and options. TCP uses sequence numbers to identify the order of data such that the data may be received out of order and reassembled.
A client establishes a TCP connection with a server through a series of messages commonly referred to as a handshake. The handshake includes the client transmitting a TCP SYN message to the server which initiates a TCP connection to the server. The server responds with a TCP SYN-ACK message which acknowledges the TCP SYN message and sets an initial sequence number (ISN) to a value chosen by the server. The client responds with a TCP ACK message that acknowledges the TCP SYN-ACK message and includes an acknowledgement number that is the ISN incremented by one. After these three messages, the TCP connection between the client and the server is established. TCP packets also include a TCP checksum which is the ones' complement sum of certain fields in the TCP header.
A fairly common distributed denial of service (DDoS) attack is known as a SYN flood that might be launched by clients (which may be participating in a botnet) that causes a high rate of incomplete TCP connections. For example, a half-open connection is a connection where the client has sent SYN message, the server has responded with a SYN-ACK message, and the server is waiting for the client to respond with an ACK message. In a SYN flood attack, malicious client(s) typically send many SYN messages to a TCP server with no intention of ever responding to the SYN-ACK message with an ACK message. The server may maintain state for all half-open connections (e.g., waiting for the client to respond with a TCP ACK message to complete the handshake) and the SYN flood may cause memory pressure for the TCP server (e.g., an overflowing connection tracking table, an overflowing SYN queue, etc), which may lead to the server failing or denying service to legitimate clients, and it may create a high interrupt rate from the network interface card on the attacked server. Thus, these incomplete TCP connections consume resources on TCP servers such as CPU time and/or memory usage.
A SYN packet can be represented in a signature format. For example, p0f is a signature format that consists of colon separated values that describes a SYN packet in a human-readable way. For instance, the signature format may include the following elements: the IP version (e.g., IPv4 or IPv6), an estimated initial TTL value, an IP options length, a maximum segment size (MSS) specified in the TCP options, a window size specified in the TCP header, a window scale specified TCP options, a TCP options layout that lists the TCP options in the order they are seen in the TCP packet (e.g., no-operation, MSS, window scaling, selective ACK permitted, selective ACK, timestamp, end of options followed by x bytes of padding), zero or more quirks (comma separated list of unusual (e.g., ACK number set in a non ACK packet) or incorrect (e.g., malformed TCP options) characteristics of a packet (e.g., “don't fragment” (DF) bit is set in IP header, DF bit is set and IP identification field is non zero, DF bit is not set and IP identification is zero, explicit congestion flag is set, reserved (“must be zero”) field in IP header is not zero, flow label in IPv6 header is non-zero, sequence number is zero, ACK field is non-zero but ACK flag is not set, ACK field is zero but ACK flag is set, URG field is non-zero but URG flag not set, URG flag is set, PUSH flag is set, timestamp 1 is zero, timestamp 2 is non-zero in a SYN packet, non-zero data in options segment, excessive window scaling factor (window scale greater than 14), match a packet sent from the Linux network stack (IP.id field equal to TCP.ts1 xor TCP.seq_num), and malformed TCP options), and payload class (TCP payload size). Each SYN packet signature may be in the form of a string with separated values such as colon separated values. The SYN packet signatures may be compared against a fingerprint database that is compiled by an expert, and is static.